Copyright © 1973 American Telephone and Telegraph Company 

The Bell System Technical Journal 

Vol. 52, No. 9, November, 1973 

Printed in U.S.A. 



A Geometric Theory of Intersymbol 
Interference 

Part II: Performance of the Maximum 
Likelihood Detector 

By D. G. MESSERSCHMITT 

(Manuscript received May 24, 1973) 

In a companion paper, 1 a geometric approach to the study of intersymbol 
interference was introduced. In the present paper this approach is applied 
to the performance analysis of the Viterbi algorithm maximum likelihood 
detector (MLD) of Forney. 2 -* It is shown that a canonical relationship 
exists between the minimum distance, which Forney has shown determines 
the performance of the MLD, and the performance and tap-gains of the 
decision-feedback equalizer (DFE). Upper and lower bounds on the 
minimum distance are derived, as is an iterative technique for computing 
it exactly. 

The performances of the MLD, DFE, and zero-forcing equalizer (ZFE) 
are compared on the yf channel representative of coaxial cables and some 
wire pairs. One important conclusion is that, previous statements not- 
withstanding, 2 - 4 even the MLD experiences a substantial penalty in S/N 
ratio relative to the isolated pulse bound on this channel of practical 
interest. 

i. intboduction 

Forney 2 ' 3 has detailed the Viterbi algorithm version of the maximum 
likelihood detector (MLD) of digital sequences in the presence of 
intersymbol interference. He asserts that the probability of bit error 
of the MLD in additive white Gaussian noise can be bounded at high 
S/N ratios in the form 

K iG (^) S P. £K.Q(*^), (i) 

where K L and K u are constants, Q is the Gaussian distribution 
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function, 

Q(s) = 4= r<r*'*dy, (2) 

V27T J* 

d m in is the minimum distance between any two transmitted signals 
(it will be denned more fully in Section 2.2), and <r 2 is the noise variance. 
For comparison purposes, the probability of error for a matched filter 
receiver in the absence of intersymbol interference is 



= «(#)■ < 3 > 



where R is the energy of an isolated pulse [(1) reduces to (3) in this 
case]. 

Forney also asserts that the lower bound of (1) is also a lower bound 
on the error probability of any receiver. 4 Thus, the MLD achieves, 
within the multiplicative constant KJK L , the minimum probability 
of error attainable by any receiver at high S/N ratios, and, in a very 
fundamental sense, the quantity 

dmln/^O 

is a measure of the effective decrease in the S/N ratio (relative to the 
detection of an isolated pulse) resulting from intersymbol interference. 

The determination of the quantity d mln (known as the "minimum 
distance problem") is therefore a very important one for, even if the 
implementation of the MLD is not contemplated on a particular 
channel, cCn is a measure of the potential performance which can be 
obtained using receivers of arbitrary complexity. Unfortunately, on 
channels with severe intersymbol interference, the exact analytical 
determination of dgun does not appear feasible because of the nonlinear 
nature of the problem. 

The minimum distance can be determined numerically by the 
"brute force" technique of calculating a sequence of converging upper 
bounds. A shortcoming of this method is that it gives no assurance 
as to when convergence to the desired accuracy has occurred. In 
addition, it gives no insight into the nature of djnjn and its relationship 
to the intersymbol interference or to the performances of other 
receivers. 

In this paper, we attack the minimum distance problem using a 
geometric theory of intersymbol interference developed in companion 
papers. 1,5 A canonical relationship will be shown between d^ and the 
decision-feedback equalizer (DFE). This relationship will be exploited 
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to derive simple lower and upper bounds on d^ in terms of the tap- 
gains of the DFE transversal filter and the S/N ratio performance of 
the DFE. In addition, an iterative procedure will be derived for the 
calculation of d£un to any desired accuracy using a sequence of con- 
verging upper and lower bounds on d^ in . The lower bounds give us a 
measure of the degree of convergence and enable us to terminate the 
calculation when the desired accuracy is assured. 

After consideration of the minimum distance problem in Section II, 
the performance of the zero-forcing equalizer (ZFE), DFE, and MLD 
is compared on a channel of practical interest in Section III. 

II. PERFORMANCE OF THE MLD 

The minimum distance problem will now receive consideration. The 
first step is to briefly review the notation of a companion paper. 1 

2.1 Notation 
The reception from a PAM communication channel takes the form 

r(t) = EB k h(t -kT) + n(t), (4) 

k 

where each B k assumes one of a finite number of predetermined values 
(the data being transmitted), h{t) is square-integrable (element of 
L 2 ),* and n(t) is white Gaussian noise. 

When we denote h(t — kT) as an element of L 2 by h k , M(h k) k G I) 
is the smallest closed linear subspace of L 2 containing all finite linear 
combinations of elements of the set {h k) k G I\- The projection of a 
vector x on M (h k , k £ /) is denoted by P[x, M(h k , k E /)]. The for- 
ward matched-filter transversal-filter combination of the DFE cor- 
responds to the L 2 inner product of the reception r(t) with the element 

ejt = h k - P[]i k , M(h m , m > fc)] (5) 

and is orthogonal to the subspace M(h m> m > k). The quantity 



where 



l e II 
/to 



Rk = (h m , h m+k ) (6) 



* We denote by L 2 the space of square integrable waveforms with inner product 
(x,y) = ] x(t)y(t)dt 



and norm ||a;|| 2 = (x,x). 
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is the effective decrease in S/N ratio relative to an isolated pulse for 
the DFE. Thus, ||e + ll 2 plays the same role for the DFE as d^ a plays 
for the MLD. 

The sequence of vectors { w k = e^/||e*M| } is an orthonormal sequence 
in Z/2, and h n has the orthogonal expansion 

00 

h n = 23 C m w m+n , (7) 

J7l=0 

where the coefficients \C m \ can be determined by the method of 
Ref. 1 for channels with either a rational or nonrational spectrum. 
In particular, we have 

C„ = ||e + l|. (8) 

Of course, it is apparent that (7) is valid only as long as \\e$\\ > 0, 
which is true if and only if a DFE exists. 

2.2 Interpretation of the Minimum Distance 

The MLD described by Forney 2 consists of a combination of a 
matched filter followed by a causal or anticausal transversal filter, 
the combination of which he calls a "whitened matched filter/' 
followed by a dynamic programming algorithm known as the Viterbi 
algorithm. 3 The whitened matched filter forms a sequence of sufficient 
statistics for the detection of the data digits and has independent noise 
samples at the output. As pointed out by Price, 8 the anticausal 
whitened matched filter is identical to the forward linear filter portion 

of the DFE. 

The signal at the output of the whitened matched filter (or DFE 
forward filter) is 1 



r k = ClB k + L C C m B k - m + n k , (9) 

where n k is a noise sample. The DFE forms the quantity 

rl = r k - t C C m 8k- m (10) 

and applies it to a decision threshold to determine the estimated digit 
8 k . The MLD detector, on the other hand, assumes that the sum in 
(9) is truncated to M terms and determines the sequence {fi k \ so as 
to minimize 

L r k - E CoC*A-J • (ID 
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Thus, the two receivers perform similar functions on the same sufficient 
statistics r„, the major differences being the greater complexity of the 
MLD and the susceptibility of the DFE to decision errors. We will 
now demonstrate the less obvious conclusion that the performance of 
the MLD is closely related to the DFE as well. 
The minimum distance, c^, n , is defined as 2 






N 

H Cnkn 
n=0 



(12) 



where the infimum is over all error sequences (e , • • •, e N ) and all N.* 
Each e* assumes the value +1, -1, or zero (for simplicity, the binary 
case with B k = 1 or is considered). Thus, d mi „ is the minimum 
distance in L 2 between two signals in the signal set. It is apparent that 

dmin ^ #o, (13) 

since R corresponds to «„ = 0, n > 0. Thus, d^/Ro, which is the 
S/N ratio penalty, is a number between zero and unity as it should be. 
It is apparent in (12) that without loss of generality we can choose 
€ = 1 and write 



dLm = inf 



AT 
n=l 



(14) 



The sum in (14) is an element of M(h k , k ^ 1), and the minimization 
in (14) is an attempt to find the element of M(h k , k ^ 1) with manifold 
coefficients ( + 1, —1, 0) which is closest (in £ 2 metric) to h . We know 
that the closest element without the restriction in coefficients is the 
projection of h on M (h k , k ^ 1), P[h Q , M(h k , k ^ 1)]. Thus, in- 
tuitively, d 2 m{D is determined by how closely the projection can be 
approximated by an element with restricted manifold coefficients. To 
formalize this intuition, add and subtract the projection from (14) and 
utilize (5), 



e+ + P[h , M(h k , k ^ 1)3 + £ e n h, 

n=l 



= ||e + l| 2 + inf 



Plho, M(h k , fc fc 1)] + £ *nh n 



71=1 



(15) 



where the fact that ef is orthogonal to M(h k , k > 0) has been used 
to eliminate the cross-product in (15). The most immediate conse- 
quence of (15) is that 

dim ^ lh + || 2 . (16) 



* In most cases of interest, the infimum will be achieved for finite N. 
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We have thus succeeded in proving formally what should be obvious 
from considerations of the relative complexity of the two receivers: 
The effective S/N ratio of the MLD always exceeds that of the DFE 
(and hence ZFE 1 ).* The second consequence of (15) is the formaliza- 
tion of our intuition through the assertion that the amount by which 
the S/N ratio of the MLD exceeds that of the DFE is governed by the 
coarseness of the best approximation to the projection by the element 
with restricted coefficients : The poorer the approximation, the better 
the S/N ratio of the MLD. 

Writing the projection in the form 



P[h , M(h k , k > 0)] = 



2^ a m tl m , 
m-1 



(17) 



we note that the a+ are the tap-gains of the DFE forward transversal 
filter, and rewrite (15) as + 

2 



ffmln — 



Ml 



+ inf 



£ fa - a+)h n 



71=1 



(18) 



Equation (18) shows the fundamental relationship between the 
minimum distance, the effective S/N ratio of the DFE (in the form 
of ||e^|| 2 ), and the tap-gains of the DFE transversal filter. In particular, 



we can assert that d^ao = 
— 1, or zero. 



|e^|| 2 if and only if the tap-gains are all +1, 



2.3 Bounds on the Minimum Distance 

Equation (18) can be used to derive bounds on d 2 ^ in terms of the 
DFE tap-gains. From the identity 1 



N 



£ (e„ - a+)h n 

n = l 



= fa - a£)* 
we immediately get the bounds 



N e — a + 

fa+T, - \ K 

„=i € fc - aj 



(19) 



E fa - o+)A, 

71=1 



, ffa " afi'Heb+ll', h = 1 
^ \ (20) 

fa - aJMM». k > i. 



*We are tempted to argue that (16) is implied by the assertion in Ref. 2 
that the MLD achieves the lowest effective S/N ratio of any receiver. However, 
that is not the case, because of the effect of decision errors on the DFE. The effective 
S/N ratio of the DFE could be higher than that of the MLD, and yet the DFE could 
have at the same time a higher error probability because of error propagation. 

f We have taken the liberty of writing a sum over infinite error sequences, where 
it is understood that the infimum is only over error sequences with a finite number 
of nonzero terms. . . 

* In (19) it is assumed that (e* - at) * 0. When e* - at = 0, (20) is trivially 

satisfied. 
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since 

<f (fn - gjj , 

„=i (e A - a£) 

is an element of M(h m , m * k). In (20), e is the ZFE filter defined in 
Ref. 1, 

e = h - P[/i , M(h h ,k 9^ 0)]. (21) 

In addition, if we define X min (iV) and X ma *(A0 as the minimum and 
maximum eigenvalues of the correlation matrix 

Rat = [i2 m _„] 1 ^ m,n ^N, 
then we can assert that 



Knin(N) £ (€n "= <*+)* ^ 



it 

£ («n — a+)h„ 



N 



A standard Toeplitz form result 7 asserts that* 



^ A mas (A0 E («• - a+)\ (22) 

n=l 



lim X niin (JV) = 7= ess inf #(o>) 

JV— 'CO ./ 

lim X max (iV) = 7n ess sup #(w). 

Applying (18), (20), and (22), we get three lower and one upper bound 
on d£ji n in terms of the tap coefficients of the DFE, 

r||e + || 2 min(ei-a 1 +) 2 

dim ^ lk + ll 2 + 1 ||co|| 2 min (e* - at) 2 , k > 1 

M 

1 X 

yp {ess infi? (&>)} lim min £ (e„ - a+) 2 

1 ^ 

^Ln ^ ||e + || 2 + ™ {ess sup #(a>)} lim min £ (e„ - a+) 2 . (23) 

x v—.-. .......... _i 



In addition, an upper bound can be obtained by substituting any error 
sequence into (IS) ; a reasonable choice is 



+ 1, a+<-i 
- 0, - § < at < h 
-1, at>\ 



(24) 



* For all practical purposes, "ess inf" and "ess sup" can be replaced by "min" 
and "max," respectively. 
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These five bounds can be useful in estimating the penalty in S/N 
ratio for the MLD. They all require the existence of a DFE and 
require that the projection can be written as the convergent sum of 
(17).* The second and third bounds of (23) are an improvement on 
(16) only when the increasingly stringent requirements that a ZFE 
exist (|jeo|| > 0) and R(u>) be uniformly bounded away from zero 
(almost everywhere) are imposed. The requirement of the upper 
bound of (23) that R(w) be uniformly upper bounded (almost every- 
where) will generally be satisfied in practice. All the bounds require 
a pointwise minimization over error sequences, a task much simpler 
than imnimizing (12) directly. 

As a simple application of these bounds, consider the exponential 
autocorrelation 



Then we have 1,2 



R k = A\ 



1, 



< A < 1. 
< A ^ \ 



[2(1 - A), \ < A < 1 
||e || 2 = (1 - A 2 ) /(I + A 2 ) 

||e + || 2 = 1 - A 2 

at = - A, at = 0, k > 1. 

The first and third bounds of (23) become 

1 - A*, < A < I 



(25) 



(26) 



d 2 mlD ^ i 



<&»£ 



(1 -A 2 ) (2 + A 2 -2A), 

1 - 2A 3 /(1 + A), 



[2(1 - A)(l + A 2 )/(1 + A), 
and the upper bound of (23) becomes 

2A 3 



!< A < 1 

< A g | 

* < A < 1 



(27) 



(28) 



«mln = 



1 + 



1 - A' 



< A ^ * 



* < A < 1. 



(29) 



2(1 - A 2 ), 

These bounds are plotted in Fig. 1. The upper bound of (24) is equal 
to d^n and is not plotted. 

*If the projection of h on P(h k , k ^ 1) cannot be written in the form of (17), 
the bounds of (22) to (24) can be fixed up by considering the projection on P(h k , 
1 ^ k g N) and taking limits as N -» « . The tap-gains will then be a function of N, 
and the process will be more difficult. 
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Fig. 1 — Bounds on dJ Jn for an exponential autocorrelation. 

The bounds just determined have the disadvantages that (i) they 
require calculation of the DFE tap coefficients and (it) they do not 
give precise results on d 2 mn . The exact value of dj^ can be determined 
numerically by the direct minimization of (12) ; by letting N — > °o 
while exhaustively minimizing over error sequences, we get a sequence 
of upper bounds on e&m which approach d*^ monotonically. The 
obvious difficulty with this method is that the number of error se- 
quences which must be checked grows as 3*, and the computational 
effort soon becomes unreasonable. What happens in practice is that 
the true minimum is achieved for a finite (and small) N. However, 
unless we have some method of determining when the true minimum 
is reached, there must always remain a degree of uncertainty as to 
whether the true minimum has been reached. 

Our approach to this computational problem will be to derive a 
sequence of lower bounds on e&m which also approach d^ lD monotoni- 
cally. We can then halt the process at a value of N where the upper and 
lower bounds are close enough to ensure knowledge of c^nn within the 
desired accuracy. To this end, we will utilize the orthogonal expansion 
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of (7). Substituting (7) into the sum of (12), 



£ Cn^n = £ «n £ C m W n+ , 

71=0 n=0 m=0 



where 



= £ n w t 

TO=0 



(8m — £ tkCm-k- 

fc=0 



Then, because the {w n | are orthonormal, 



£ «"/in 

71=0 



= £ /«. 

rt=0 



(30) 



(31) 



(32) 



It appears that we may have made life more difficult for ourselves, 
because even when we substitute a finite sum on the left of (32) we 
must still evaluate an infinite sum on the right. However, note that 
since the terms in the sum are positive, 



£ e n h n 

n=0 



N 

^ £ & 

n=0 



(33) 



where the sum on the right is always finite and is in terms of a finite 
length error sequence (eo, ■ ■ ■, £n)- Hence, 



N 

<4i n i? min £ p* 

ci, •••,€» n=0 

£0=1 



(34) 



and, furthermore, the right side of (34) approaches the left side 
monotonically as N — > » . 

The minimization of (34) is no more or less difficult to perform than 
that of the direct minimization of (12). It does require the existence 
of a DFE and evaluation of the coefficients { C m \ . A reasonable pro- 
cedure is, at each stage of N, to minimize the right side of (34) to 
obtain a lower bound on d^ and substitute the minimizing sequence 
into (12) to obtain the upper bound* on d^ n . When the lower and 
upper bounds are sufficiently close, the process can be terminated. 



* Note that any sequence substituted into (12) yields an upper bound on d£, in , 
and the one which minimizes (34) is as good as any. On the other hand, only the 
sequence which minimizes (34) yields a valid lower bound, so it must be minimized. 
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The minimization of (34) can be assisted slightly by dynamic 
programming. Defining 

N 

f N - m (e 1 ,---,e m - 1 )= min £ 0|, (35) 

tm, • • \«.v n=m 



we note that 



N 

min £ ft = min [/^(ei) + 0?] (36) 



«!,■••,«# n = 1 

with a recursion relation for js-m («i, • • •, e m _i), 
/Ar_ m+ i(ei, •••,e m _ 2 ) = min £ ft 

(m-ii • • • ,ew n=m— 1 



min £ « + 0! 



-] 



= min 

= min [/^_ m ( ei . • . €b ,_0 4- ft_J. (37) 

tm-l 

Because there is no possibility of using forward dynamic programming 
in this case, the savings in computation for this method is not too 
spectacular. Each /3„ must still be evaluated for 3^ error sequences; 
the savings is in eliminating the need for summing ft for most of the 
combinations of 3^ error sequences. 

We note in passing that using the FFT algorithm to reduce the 
computational effort in the convolutional sum of (31) is a possibility. 
However, the 3" sequences for which it must be evaluated becomes a 
limiting factor long before the savings of that method becomes 
substantial. 

In the foregoing discussion, the existence of a DFE has been re- 
quired [that is, ||e^|| > or equivalently log R(u) is integrable, where 
R(u) is the equivalent power spectrum of the channel 1 ]. When log R (w) 
is not integrable (as when it vanishes on an interval), there does not 
appear to exist an expansion of the type (31) to (32). What can be 
done is to use the Gram-Schmidt expansion of the form 

h m = £ (h m , w k )w k , (38) 

where w k is the orthonormal sequence obtained from {h k } by the usual 
Gram-Schmidt orthonogalization procedure. This expansion merely 
requires that { h k \ be linearly independent, which is guaranteed by the 
existence of an interval on which R(u) does not vanish. 1 From (38), it 
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follows that 

oc oo m 

L Cmh m = £ € m Y, (km, W k )w k 

m =0 m=0 *=0 

= £ /to (39) 

0* = £ € m <A m , w*). (40) 

m=fc 

The key point is that the summation in (40) is infinite, so that evalua- 
tion of the lower bound of (34) is now necessarily over infinite error 
sequences. The finite sum in (31) results from the form of the expan- 
sion (7) in which h n is expanded in terms of all future w k 's, and this 
expansion is in turn dependent on h n not being an element of 
M(h k , k > n). Thus, when a DFE does not exist there appears to be 
no alternative to evaluating a sequence of upper bounds to d^ n ob- 
tained by a finite sum approximation without the benefit of lower 
bounds to measure the degree of convergence. 

III. THE PERFORMANCE OF THREE RECEIVERS ON THE V/ CHANNEL 

Results of a calculation of the performance of the MLD, DFE, and 
ZFE will now be reported for the V/ channel, for which the attenuation 
in decibels increases as the square root of frequency. The V/ channel 
is a good approximation to coaxial cable, as well as to some cables 
consisting of wire pairs, and for this reason it is of great practical 
interest. 

Many present high-speed digital transmission systems use some 
form of linear equalization, and their performance will be reasonably 
well approximated by that of the ZFE. Thus, the comparison between 
the ZFE and the MLD gives us an indication of the size of the gap in 
performance between common transmission systems in use today and 
what could theoretically be achieved by much more complex receiver 
designs. * The comparison with the DFE is much less interesting, be- 
cause the susceptibility of the DFE to decision errors is not included 
in the present analysis and, as will be shown shortly, is of such a 
magnitude on the V/ channel as to essentially invalidate the perfor- 
mance estimate we calculate. 



* This comparison is, of course, very idealized. The only impairment we consider 
is additive Gaussian noise. 
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■ACTUAL 
•APPROXIMATION 




ZFE 



30 40 50 60 70 

y = LOSS AT HALF BAUD RATE IN DECIBELS 

Fig. 2 — Performance of three receivers on the -ff channel. 

The power spectrum of the V/ channel is given by 

| H{u) | 2 = 2irKW e- 2K ^, (41) 

where H(co) is the frequency response of the channel and K is a 
parameter proportional to the line length. The usual convention is to 
designate the loss at the half-baud rate (w = w/T), 



H 



7 = - 10 log 



(?) 



in which case 



H(0) 



(dB), 



K = 



■k 20 log e 



(42) 
(43) 



The effective penalties in S/N ratio relative to the isolated pulse 
bound can be calculated for the ZFE and DFE using the methods of 
Ref. 1, and for the MLD using the methods developed in Section 
II. The result is shown in Fig. 2 for the range of y of practical interest. 
Most high-speed transmission systems in use today have a y less than 
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about 65 dB because of limitations in the maximum gain which can 
be incorporated into a repeater without excessive coupling of the 
output back into the input. 

One interesting feature of Fig. 2 is that even the MLD has a sub- 
stantial S/N ratio penalty (15 dB) on the V/ channel. Thus, Forney's 
statement 3 that on most channels intersymbol interference does not 
have to lead to a significant degradation in performance does not apply 
to channels with very severe intersymbol interference, such as are 
commonly used in high-speed transmission systems. 

The value of (&,„, valid for Fig. 2, as well as many other examples 
considered by this author and Forney, 4 is 

dU = 2(i?o - Ri), (44) 

where R k is the autocorrelation of the received pulse. * An approxima- 
tion to (44) valid for large y is derived in Appendix A and plotted in 
Fig. 2 as a dotted line. Approximations to the S/N ratio penalty of 
the ZFE and DFE are also derived in Appendix A and plotted in Fig. 
2. An intuitive interpretation of eq. (44) is given in Appendix B. 

As an illustration of the speed of convergence of (34), the sequence 
of upper and lower bounds is illustrated in Fig. 3 for a yf channel with 
7 = 60 dB. These bounds are within 1 dB for N = 1 and 0.5 dB for 
N = 3. Thus, convergence is very rapid, even for severe intersymbol 
interference. 

A word of caution is in order with respect to the curve for the DFE 
in Fig. 2. This curve does not take into account the effect of decision 
errors on the performance of the receiver. The DFE subtracts, prior to 
the decision threshold on data digit B k , the quantity 

£ M*-«, (45) 

m=l 

where & k - m is the receiver's previous decision on B k - m and b m is the 
tap-gain of the DFE feedback filter. The resulting quantity which is 
applied to the threshold is 1 

boB k + £ b m {B k _ m - B k . m ) + n k , (46) 

where n k is a noise sample. Whenever the b m 'a are large with respect 
to b , a single decision error will likely cause many more errors. The 



* This corresponds to the error sequence (1, —1, 0, 0, • • ■) or, in the notation of 
Forney, (1 - D). 
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-LOWER BOUND ON d. 



-UPPER BOUND ON d M ,M 2 




2 3 4 5 6 7 

N ■ NUMBER OF TERMS IN ERROR SEQUENCE 



10 



Fig. 3 — Convergence of lower and upper bounds on c/2, ln (V/ channel with y = 60 dB). 

coefficients of (46), given by (9), are tabulated in Table I for several 
values of 7. 

Needless to say, the situation is hopeless for the large 7 ; the effect 
of a single decision error will be major and will last for a long time. 
Even for 7 = 20, the reduction in noise margin resulting from a pre- 



Table I — Coefficients of the DFE Feedback Filter (6 m ) 



m 


fem 




7 = 20 


7 = 40 


7 = 60 





1 


1 


1 


1 


0.61 


1.4 


2.2 


2 


0.36 


1.3 


2.8 


3 


0.25 


1.1 


2.9 


4 


0.18 


0.94 


2.9 


5 


0.14 


0.80 


2.8 


10 


0.06 


0.42 


1.9 


47 


0.006 


0.06 


0.38 


174 


0.001 


0.009 


0.06 
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vious decision error will be significant for five or ten subsequent 
decisions. We must conclude, then, that Fig. 2 will not be representa- 
tive of the true performance of the DFE, and further that the DFE 
may not be a suitable receiver for the V/ channel*. 

In terms of repeater spacing and baud rate, Fig. 1 can be interpreted 
in two ways. If the ZFE is replaced by an MLD, the same level of 
performance can be maintained while either increasing the repeater 
spacing with a constant baud rate or increasing the baud rate with 
the same repeater spacing. To illustrate this, consider the example of a 
ZFE operating at a given level of performance on a VJ channel with 
y = 40 dB. Then 7 can be increased to 60 dB at the same effective 
S/N ratio. This corresponds to a 50-percent increase in repeater spac- 
ing at a constant baud rate (since 7 goes up linearly with the repeater 
spacing). However, since the repeater spacing has increased, the trans- 
mitted power must also be increased by 3.5 dB to maintain a constant 
isolated pulse energy at the receiver. 1 

If the repeater spacing is held constant, an increase in baud rate by a 
factor of (1.5) 2 , or 125 percent, will also result in a 50-percent increase 
in 7. Here too, the average (but not peak) transmitted power is in- 
creased by 3.5 dB. 

The conclusion of these results is that there is a fairly large gap be- 
tween the performance of linear equalizers and the theoretical limit 
on the V/ channel. It is probably fair to say, however, that practical 
constraints on repeater complexity, speed of operation, and gain makes 
the attainment of a substantial portion of this potential improvement 
on high-speed transmission systems very difficult, at least for the 
present. Such is not the case for low-speed applications, such as voice- 
band data, where the implementation of the MLD can be contemplated 
on the basis of existing technology. 

IV. CONCLUSIONS 

In this paper, the minimum distance measure has been interpreted 
geometrically, related to equalization (the decision-feedback equalizer 
in particular), and bounded in several ways. A practical numerical 
technique has been developed for calculating the minimum distance 
without considering unnecessarily long error sequences. 

* Tomlinson 8 has invented a method of avoiding the error propagation problem by 
subtracting out interference from past data digits in the transmitter. 

+ The received pulse energy is proportional to y~ 2 , so that the peak and average 
transmitted power must be increased by 20 log (60/40) = 3.5 dB. 
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Numerical results for the V/ channel reveal that the penalty in S/N 
ratio relative to the isolated pulse bound for the MLD can be sub- 
stantial for this channel, and that the gap in performance between the 
MLD and linear equalization is also substantial. The latter suggests 
that further attempts at finding receivers without the complexity of 
the Viterbi algorithm MLD but which nevertheless improve on the 
performance of linear equalization might well be fruitful. The decision- 
feedback equalizer does not appear to fit this bill because of its serious 
error propagation problem when confronted with intersymbol inter- 
ference as severe as that found on the ^[f channel. 

APPENDIX A 

Autocorrelation of the V/ Channel 
From (41), the autocorrelation is 

R k = - I" I H(a) | 2 cos (u>kT)da> 

TT JO 

I x exp J = x J cos (kx 2 )dx. (47) 



4K 2 fl r 



T 



(2K \ 
t= x J and dv = x cos (kx 2 )dx, 

we get 

which is given in terms of the Fresnel Integral, 9 
p g jPg, [[I r ( K /2\1 (K*\ 

+G--(W)]-(S)1. » 

where 



C(x) = J* \ob ( \ A dy 
8(x) = J* a* (llf) dy . 



An accurate approximation to Ri valid for large 7 is easily obtained 
from (47) by substituting the first two terms of a Taylor series for 
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COS X 2 , 



fli 2 /3 2 fl r x(\-^\ <r<"dx 

= *o(l-f 4 ), (49) 



where 



Hence 



and 



3 = vF 



2(i2 — Ri) = -~et 



- 10 log 2(Ro p fil) S 40 log t - 56.2. (50) 

/to 

Approximations to ||e || 2 and ||e^"|| 2 can also be derived by assuming 
that H(u) =0, | co | > t/T, or equivalent^ that |#(co)| 2 = R(u). 
The resulting S/N ratio penalties are 

- 10 log ||eo|| 2 /flo ^ 7 + 25.15 - 30 log 7 (51) 

- 10 log ||e + H7-Ro S f7 + 15.76 - 20 log 7 • (52) 

Equations (50) to (52) are plotted in Fig. 2 as dotted lines. 

APPENDIX B 

Interpretation of Equation (44) 

It is straightforward to show that whenever 

f- 1 ^ 0.5 (53) 

/to 

we have 

<&in ^ 2(fl - #i) ^ Bo- (54) 

Noting that 

Ri = (/io, Ai) = ||M IIM cos 6 
= R cos 5, 

where 8 is the angle between h and fei, eq. (53) becomes 

6 ^ 60°. (55) 

The geometric interpretation of (55) is shown in Fig. 4, where it is 
seen that (54) is satisfied until 9 = 60°, when the triangles become 
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Fig. 4 — Geometric interpretation of eq. (44). 

equilateral. As long as (55) is satisfied, h — hi is a shorter vector than 
ho. 

In the case of the vf channel, Ri/Ro is very close to unity. Thus, 
h — hi is a very short vector. Although it will certainly not always be 
the case, a plausible explanation for the fact that longer error sequences 
do not yet yield a shorter vector is that the addition of other translates 
of hk (such as ±h 2 ) adds further components in other dimensions. 
Presuming that it does not reduce the component in the h — hi plane, 
it can then only increase the length of the total vector. 
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